Input devices for augmented reality applications

ABSTRACT

A new interaction/input device for an augmented reality system is provided. In an augmented reality system using infrared video-based tracking, the interaction/input device is employed by placing markers, e.g., small disks, at a predetermined location in a scene viewed by a user which are augmented to simulate physical buttons. These augmented markers, as viewed through the augmented reality system, can then be physically manipulated by the user. The user will put their fingers on one of these markers, and in turn, the infrared video-based tracker will recognize this action and process it accordingly. The augmented reality system can also augment simulated menus in the user&#39;s view giving the user the necessary feedback for interaction. By exploiting conventional tracking technology, the interaction/input device can be implemented with minimal additional hardware and minimal additional processing required by the augmented reality system.

PRIORITY

This application claims priority to an application entitled “NEW INPUTDEVICES FOR AUGMENTED REALITY APPLICATIONS” filed in the United StatesPatent and Trademark Office on May 31, 2001 and assigned Ser. No.60/294,850, the contents of which are hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to augmented reality systems,and more particularly, to input devices and methods for user interactionwith an augmented reality system.

2. Description of the Related Art

Augmented reality is the technology in which a user's view of the realworld is enhanced with additional information generated from a computermodel, i.e., the virtual. The enhancements may include labels, 3Drendered models, or shading and illumination changes. Augmented realityallows a user to work with and examine the physical world, whilereceiving additional information about the objects in it. Some targetapplication areas of augmented reality include computer-aided surgery,repair and maintenance, facilities modification and interior design.

In a typical augmented reality system, the view of a real scene isaugmented by superimposing computer-generated graphics on this view suchthat the generated graphics are properly aligned with real-world objectsas needed by the application. The graphics are generated from geometricmodels of both virtual objects and real objects in the environment. Inorder for the graphics and the real-world objects to align properly, thepose (i.e., position and orientation) and optical properties of the userand virtual cameras must be the same. The position and orientation ofthe real and virtual objects in some world coordinate system must alsobe known. The locations of the geometric models and virtual cameraswithin the augmented environment may be modified by moving its realcounterpart. This is accomplished by tracking the location of the realobjects and using this information to update the correspondingtransformations of the geometric models within the virtual world. Thistracking capability may also be used to manipulate purely virtualobjects, ones with no real counterpart, and to locate real objects inthe environment. Once these capabilities have been brought together,real objects and computer-generated graphics may be blended together,thus augmenting a dynamic real scene with information stored andprocessed on a computer.

Recent advances in both hardware and software have made it possible tobuild augmented reality (AR) systems that can run on regular desktopcomputers with off-the-shelf display and imaging devices. For example,the ARBrowser™ system, developed at Siemens Corporate Research, runs ona 400 MHz Pentium III machine. The system uses infrared video-basedtracking technology, also developed at Siemens Corporate Research. Thistracking technology considerably reduces the time required for trackingand pose estimation while maintaining the robustness and accuracy of thepose estimation. These advances allow the AR system to run on a regularcomputer without specialized display hardware at full frame rate,currently 30 frames per second (fps).

As the above-described advances in tracking speed and accuracy helpedrealize real-time augmentation, user interaction issues have become morevisible. To fully realize the potential of AR systems, users need tointeract with the systems and conventional methods, such as a keyboardand mouse, have proved to be very cumbersome. More advanced methods ofinteraction, i.e., speech driven methods, are hard to integrate with ARsystems due to their inherent difficulties, such as “training” thespeech driven system, and their large processing power requirements,which will hinder the running of the AR system resulting in lower framerates and additional delays. Most augmented reality systems, currentlybeing used and developed, are lacking easy-to-use, intuitive andeffective means of interaction with the user.

SUMMARY OF THE INVENTION

It is therefore an object of the present invention to provide aninteraction/input device for an augmented reality (AR) system.

It is another object of the present invention to provide an input deviceand method for use for an augmented reality (AR) system which iseasy-to-use and intuitive for a user, thus enhancing the experience ofthe user with the system.

It is a further object of the present invention to provide aninteraction device which requires minimal additional hardware andminimal additional processing power.

To achieve the above and other objects, a new interaction/input devicefor an augmented reality system is provided. By exploiting conventionaltracking technology, the interaction/input device can be implementedwith minimal additional hardware and minimal additional processingrequired by the augmented reality system. In an augmented reality systemusing infrared video-based tracking, the interaction/input device isemployed by placing markers, e.g., small disks, at a predeterminedlocation in a scene viewed by a user which are augmented to simulatephysical buttons. These augmented markers, as viewed through theaugmented reality system, can then be physically manipulated by theuser. The user will put their fingers on one of these markers or disks,and in turn, the infrared video-based tracker will recognize this actionand process it accordingly. The augmented reality system can alsoaugment simulated menus in the user's view giving the user the necessaryfeedback for interaction.

According one aspect of the present invention, a system for augmenting auser's view of real-world objects with virtual objects to provide acomposite augmented reality image is provided. The system including adisplay device for displaying the composite augmented reality image tothe user; a video-based tracking system for locating real-world objects;a processor for determining the position and orientation of the user'sview based on the location of the real-world objects and for projectingthe virtual objects onto the display device; and an input deviceincluding at least one marker placed at a predetermined location in thereal world, wherein the tracking system locates the input device and theprocessor determines its functionality based on its location.

According to another aspect of the present invention, in a system foraugmenting a user's view of real-world objects with virtual objects toprovide a composite augmented reality image, the system including adisplay device for displaying the composite augmented reality image tothe user, a video-based tracking system for locating real-world objects,and a processor for determining the position and orientation of theuser's view based on the location of the real-world objects and forprojecting the virtual objects onto the display device, an input deviceis provided including at least one marker placed at a predeterminedlocation in the real world, wherein the tracking system locates theinput device and the processor determines its functionality based on itslocation.

According to a further aspect of the present invention, in a system foraugmenting a user's view of real-world objects with virtual objects toprovide a composite augmented reality image, the system including adisplay device for displaying the composite augmented reality image tothe user, a video-based tracking system for locating real-world objects,and a processor for determining the position and orientation of theuser's view based on the location of the real-world objects and forprojecting the virtual objects onto the display device, a method forinteracting with the system is provided. The method includes the stepsof providing an input device including at least one marker placed at apredetermined location in the real world; capturing video of the realworld by the video-based tracking system; analyzing the captured videoto determine if the at least one marker is visible; if the at least onemarker is visible, determining the real world location of the at leastone marker; and loading the input device's functionality into the systemto be available for the user to interact with the system. The methodfurther includes the steps of determining if the at least one marker isnot visible after entering an input mode; and if the at least one markeris not visible, performing a function associated with the at least onemarker.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features, and advantages of the presentinvention will become more apparent in light of the following detaileddescription when taken in conjunction with the accompanying drawings inwhich:

FIG. 1A is a schematic diagram illustrating an augmented reality systemwith infrared video-based tracking in accordance with the presentinvention;

FIG. 1B is a perspective view of an infrared tracker camera withinfrared illumination LEDs;

FIG. 2 is a flowchart illustrating a method of interacting with anaugmented reality system employing an input device of the presentinvention; and

FIG. 3 illustrates several views of a user interacting with theaugmented reality system in accordance with the present invention, wherecolumn 1 represents real-world views, column 2 represents views as seenfrom the infrared tracker camera and column 3 represents augmented viewsof a user.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments of the present invention will be describedhereinbelow with reference to the accompanying drawings. In thefollowing description, well-known functions or constructions are notdescribed in detail to avoid obscuring the invention in unnecessarydetail.

Generally, an augmented reality system includes a display device forpresenting a user with an image of the real world augmented with virtualobjects, a tracking system for locating real-world objects, and aprocessor, e.g., a computer, for determining the user's point of viewand for projecting the virtual objects onto the display device in properreference to the user's point of view.

Referring to FIG. 1A, an exemplary augmented reality (AR) system 10 tobe used in conjunction with the present invention is illustrated. The ARsystem 10 includes a head-mounted display (HMD) 12, an infraredvideo-based tracking system 14 and a processor 16, here shown as adesktop computer. For the purposes of this illustration, the AR system10 will be utilized in a specific workspace 18 which includes markerboard 20 and an input device 22 of the present invention.

Referring to FIGS. 1A and 1B, the tracking system used in conjunctionwith marker board 20 determines the position and orientation of a user'shead and subsequently a scene the user is viewing. Generally, theinfrared video-based tracking system 14 includes a camera 24 with aninfrared-filter lens 26 and a plurality of infrared illuminationlight-emitting diodes (LEDs) 28 mounted around the lens 24; a videocapture board mounted in the processor 16; and a set of retroreflectivemarkers, e.g., a circular disk or square tile. Video obtained from thecamera 24 through the capture board is processed in the processor 16 toidentify the images of the retroreflective markers. Because the videocaptured is filtered, the only visible items will be the onescorresponding to the retroreflective markers, i.e., items reflectinglight in an infrared frequency. Since the location of the markers areknown within a specific workspace, the processor can determine the poseof the user.

In the AR system of FIG. 1A, the marker board 20 is utilized fordetermining the pose of the user. The marker board 20 includes ten smallretroreflective disks 30 surrounded by four thin retroreflective bands32. The exact configuration and location of the marker board 20 isstored in the processor so when the camera 24 of the tracking system 14encounters the marker board 20, the pose of the user can readily bedetermined. This pose estimation is used in turn for augmentation. Thebands 32 surrounding the ten disks 30 robustify the tracking process andallow the addition of other retroreflective markers as input devices.

Once the marker board 20 is identified in the video as captured bycamera 24, the position of the input device 22 can be estimated in thevideo. The input device 22 is determined based on its physical locationin the real world in relation to the physical location of the markerboard 20. Through the tracker and pose estimation as determined by theprocessor, the position of any world point in the user's view can beestimated. This means that the projection or position of the inputdevice 22 in the user's view can be calculated, and thus, it can bedetermined if the input device is visible to the user. Once the inputdevice becomes visible, the input device's functionality can be loadedand the AR system 10 can go into a menu/input mode and wait for theuser's actions for some input events. The AR system 10 will determine ifa user is interacting by determining if a marker of the input device 22is visible or not. If the marker is not visible, e.g., by the action ofthe user covering the marker, the system will determine the marker isactivated and perform an associated function.

It is to be understood the type and functionality of an input device ofthe present invention is determined by the processor based on the knownspecific locations of markers placed in the physical world in relationto the marker board. Therefore, once the position of the marker board 20is estimated, any number of input devices can be realized by placingmarkers at known locations in relation to the marker board. For example,a single marker can be placed near the upper left hand corner of themarker board and can be augmented to represent an on/off button.Additionally, a 4×3 matrix of markers can be placed at a specificlocation in relation to the marker board to simulate a numerical keypadinput device, like those used on a telephone. Furthermore, a combinationof one or more input devices may be placed around the workspace at onetime simulating different options to the users and making the AR systemscalable.

An illustration of how a user interacts with an AR system employing aninput device of the present invention will be described below inconjunction with FIGS. 2 and 3, where FIG. 2 is a flowchart illustratinga method of interacting with the augmented reality system employing aninput device of the present invention and FIG. 3 illustrates severalviews of the user interacting with the augmented reality system inaccordance with the present invention, where column 1 represents realworld views, column 2 represents views as seen from the infrared trackercamera 24 and column 3 represents augmented views of the user and therows of FIG. 3 represent different interactions of the user.

The first row in FIG. 3 shows the AR system entering a menu/input mode.The first view illustrates a real world view of the marker board 20 withinput device 22 in close proximity at a specific, predetermined locationin the real world (Step S1). The second view of the first row is a viewof the marker board 20 and input device 22 captured through theinfrared-filtered camera 24, wherein all retroreflective markers arevisible (Step S2). Through the use of the tracking system and processor,the AR system is able to determine the three markers 34 of the inputdevice are outside the four bands of the marker board 20 and thus is theinput device 22 and it is in the user's view (Step S3). Once the pose ofthe user and input device is determined (Step S4), the AR system willaugment the user's view as in the third view of the first row. Here, asin Step S5, the three markers 34 are augmented with computer-generatedgraphics to simulate buttons or menus, e.g., the first markers reads “H”for help, the second marker reads “P” for previous page, and the thirdmarker reads “N” for next page, and thus, the AR system enters themenu/input mode (Step S6). Optionally, the AR system will overlay agraphic on the marker board 20, such as a manual to assist the user inperforming an operation to a piece of equipment in the user's view.

The second row of FIG. 3 shows the user choosing an action, i.e.,requesting the next page of the manual. The first view of the second rowshows the user's finger covering the third marker of the input device22. Camera 24 determines that the third marker is not visible, as shownin the second view of the second row and in Step S7, and the AR systemrealizes the user has prompted the system to go to the next page. Thethird view of the second row illustrates the user interacting with thesystem as seen through the HMD 12, where the user initiates an action by“pressing” the “N” button and the system performs the associatedfunction by going to Page 2 of the manual (Step S8).

The third row of FIG. 3 illustrates the that the input device 22 can beaugmented with menus in addition to buttons. In the first view of thethird row, the user places their finger on the first marker whichcorresponds to the “H” or Help button. Once the AR system determines theuser has requested help, the AR system will augment the user's view byinserting a graphic help menu with several help options, as shown in thethird view of the third row. In addition, up and down arrows will beplaced above the second and third markers during the help mode to assistthe user in selecting the help option desired. It is to be understoodthat the up and down arrows are only augmented in the user's view duringthe help mode. It is also to be understood that whenever a single markeris activated the remaining two markers can be augmented to reveal otheroptions of the activated marker.

New input devices and interaction methods for augmented realityapplications that exploit the recent advances in augmented realitytechnologies have been described. In particular, for the augmentedreality systems that use an infrared video-based tracking system, theinteraction/input devices and methods of the present invention provideintuitive, easy-to-use means of interacting with the augmented realitysystem. The system gives the user visual feedback in forms ofaugmentation, e.g., menus, to facilitate the interaction.

The input devices of the present invention do not put any additionalburden on the running or processing of the augmented reality applicationsince the AR system is already determining locations of markers fortracking purposes. The tracking system intelligently can decide if theuser is in the input/interaction mode by determining if the user islooking at the various menu markers in the scene. Furthermore, use ofvisual feedback assists the user and enhances his/her experience withthe augmented reality system greatly.

While the invention has been shown and described with reference tocertain preferred embodiments thereof, it will be understood by thoseskilled in the art that various changes in form and detail may be madetherein without departing from the spirit and scope of the invention asdefined by the appended claims. For example, the input device can beemployed in various types of AR systems, such as optical see-through orvideo see-through systems. Additionally, the input device of the presentinvention can be used in conjunction with different types of displaydevices, e.g., a computer monitor, video-capable mobile phone, personaldigital assistant (PDA), etc.

1. A system for augmenting a user's view of real-world objects withvirtual objects to provide a composite augmented reality image, thesystem comprising: a display device for displaying the compositeaugmented reality image to the user; a video-based tracking system forlocating real-world objects; a processor for determining the positionand orientation of the user's view based on the location of thereal-world objects and for projecting the virtual objects onto thedisplay device; and an input device including at least one marker placedat a predetermined location in the real world, wherein the trackingsystem locates the input device and the processor determines itsfunctionality based on its location, wherein the input device furthercomprises a plurality of markers, each of the plurality of markers beingaugmented with different virtual objects, and wherein when one of theplurality of markers is activated the remaining markers are augmentedwith virtual objects corresponding to options of the activated marker.2. The system as in claim 1, wherein the video-based tracking system isan infrared video-based tracking system.
 3. The system as in claim 2,wherein the infrared video-based tracking system comprises: a camera forcapturing video of the real world, the camera including aninfrared-filter lens for capturing light in an infrared frequency; and aplurality of infrared illumination light-emitting diodes (LEDs) forproviding the infrared light.
 4. The system as in claim 2, wherein theat least one marker of the input device is retroreflective.
 5. Thesystem as in claim 2, further comprising a marker board including aplurality of retroreflective markers surrounded by a continuous band ofretroreflective material, the marker board being located in the realworld at a known location to serve as a point of reference for theinfrared video-based tracking system.
 6. The system as in claim 1,wherein the at least one marker of the input device is augmented on thedisplay device with a virtual object representing a button.
 7. Thesystem as in claim 1, wherein the at least one marker of the inputdevice is augmented on the display device with a virtual objectrepresenting a menu of selectable options.
 8. In a system for augmentinga users view of real-world objects with virtual objects to provide acomposite augmented reality image, the system including a display devicefor displaying the composite augmented reality image to the user, avideo-based tracking system for locating real-world objects, and aprocessor for determining the position and orientation of the user'sview based on the location of the real-world objects and for projectingthe virtual objects onto the display device, an input device comprising:at least one marker placed at a predetermined location in the realworld, wherein the tracking system locates the input device and theprocessor determines its functionality based on its location, furthercomprising a plurality of markers, each of the plurality of markersbeing augmented with different virtual objects, and wherein when one ofthe plurality of markers is activated the remaining markers areaugmented with virtual objects corresponding to options of the activatedmarker.
 9. The input device as in claim 8, wherein the video-basedtracking system is an infrared video-based tracking system.
 10. Theinput device as in claim 9, wherein the infrared video-based trackingsystem comprises: a camera for capturing video of the real world, thecamera including an infrared-filter lens for capturing light in aninfrared frequency; and a plurality of infrared illuminationlight-emitting diodes (LEDs) for providing the infrared light.
 11. Theinput device as in claim 9, wherein the at least one marker isretroreflective.
 12. The input device as in claim 9, wherein theinfrared video-based tracking system comprises a marker board includinga plurality of retroreflective markers surrounded by a continuous bandof retroreflective material, the marker board being located in the realworld at a known location to serve as a point of reference for theinfrared video-based tracking system.
 13. The input device as in claim8, wherein the at least one marker is augmented on the display devicewith a virtual object representing a button.
 14. The input device as inclaim 8, wherein the at least one marker is augmented on the displaydevice with a virtual object representing a menu of selectable options.15. In a system for augmenting a user's view of real-world objects withvirtual objects to provide a composite augmented reality image, thesystem including a display device for displaying the composite augmentedreality image to the user, a video-based tracking system for locatingreal-world objects, and a processor for determining the position andorientation of the user's view based on the location of the real-worldobjects and for projecting the virtual objects onto the display device,a method for interacting with the system comprising the steps of:providing an input device including at least one marker placed at apredetermined location in the real world; capturing video of the realworld by the video-based tracking system; analyzing the captured videoto determine if the at least one marker is visible; if the at least onemarker is visible, determining the real world location of the at leastone marker; and loading the input device's functionality into the systemto be available for the user to interact with the system, wherein theloading step further comprises the step of entering an input mode, andfurther comprising the steps of: determining if the at least one markeris not visible after entering the input mode; and if the at least onemarker is not visible, performing a function associated with the atleast one marker, wherein the input device comprises a plurality ofmarkers, each of the plurality of markers being augments with differentvirtual objects, further comprising the step of, when at least one ofthe markers is activated, augmenting the remaining markers with virtualobjects corresponding to options of the activated marker.
 16. The methodas in claim 15, wherein the video-based tracking system is an infraredvideo-based tracking system and the at least one marker of the inputdevice is retroreflective.
 17. The method as in claim 15, furthercomprising the step of augmenting the at least one marker of the inputdevice with a virtual object.
 18. The method as in claim 15, furthercomprising the step of augmenting the at least one marker of the inputdevice with a virtual object representing a menu of selectable options.19. The method as in claim 15, further comprising the step of providinga marker board, the marker board being located at the real world in aknown location to serve as a point of reference for the video-basedtracking system.
 20. A system for augmenting a user's view of real-worldobjects with virtual objects to provide a composite augmented realityimage, the system including a display device for displaying thecomposite augmented reality image to the user, a video-based trackingsystem for locating real-world objects, and a processor for determiningthe position and orientation of the user's view based on the location ofthe real-world objects and for projecting the virtual objects onto thedisplay device, a method for interacting with the system comprising thesteps of: providing an input device including at least one marker placedat a predetermined location in the real world; capturing video of thereal world by the video-based tracking system; analyzing the capturedvideo to determine if the at least one marker is visible; if the atleast one marker is visible, determining the real world location of theat least one marker; and loading the input device's functionality intothe system to be available for the user to interact with the system,wherein the input device comprises a plurality of markers, each of theplurality of markers being augments with different virtual objects,further comprising the step of, when at least one of the markers isactivated, augmenting the remaining markers with virtual objectscorresponding to options of the activated marker.